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!I!ie re^rd shops of our nation have^ for nany years^ stocked and ad- 
vertised fcfreign la n gu a ge courses hopefully entitled with variant renditions 
of the self -instructional label. Ifost of us have learned to our dismay 
that a ten or twenty-dollar hill does not purchase usable language skills 
by taking such courses^ except in rather rare circumstances. 



That this same situation is repeating itself in the case of the new 
programmed^ self-instructional foreign language courses is easily seen by 



exami^ng^the book entitled Programs *6.3t A Guide to Erogrammed Instructional 



Materials coispiled and produced by the Center for Programmed Instruction 
^d The Office of Education. It will be seen that few, if any, of the 
language courses described provide any evidence of training effectiveness. 

TOe general lack of adequate evaluational data for most of the self-instructional 
foreign la ngua ge courses presently on the market is a serious deficiency that 
"the la n guage programming movement as a whole. 



The purpose of this paper is to list and describe the various criteria 
that should be. used in the evaluation of self »-instructional language courses 
so that the ^consumer, whoever he may be, does not have to purchase a "pig 
in the poke" when he buys a self -instructional foreign language course. 



In order that this be no mere academic exercise, I shall describe an 
actual example of evaluating a self -instructional foreign language course 
with the re^ization that not all the criteria here used are applicable to 
the evaluation of every other self -instructional foreign language course. 
However, it should be incumbent upon the evaluator of any other such courses 
0 give some reason why any of the criteria listed here are not utilized or 
to have his evaluation considered as incomplete. 



I shoiild then like to describe procedures Just recently csLrried. out in. 
evaluating a self-instructional foreign language course at the TTtimfl.n Resources 
ese^ch Office in Alexandria, Virginia. This course was a newly revised 
Russian self-instructional course aimed at providing the student with the 
ability to speak and understand enough Russian to obtain tactical information 
^rom newly captured prisoners of war. The original course is described in 
HumRRO Research Report No. 9 entitled "Development and Evaluation of Training 
Methods for the Rapid Acquisition of Language Skills." 



^^pgrams *63, A Guide to P rogrammed In structional Mater ials , The Center for 
Programed Instruction, Inc. , 'Washington, D.CV: U. S. Government Printing 
Office, 1963. 
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As most of us Icnow, evaluation of a self- instructional foreign 
angu^e course really begins not when the cdurse has been administered 
for the first time, but before it has been constructed. With that as 
our starting point, let me describe this course, how it was evaluated 
and results of the ' evaluation. 



This course was constructed to fill a specific need. The soldier 
who captures a PCW usually cannot speak or understand enough of the enemy 
lan^age to question his prisoner. Information concerning the immediate 
combat situation that might be of value cannot, therefore, usually be 

acquired by the man who needs it the most and who is in the best position 
to obtain it. 



+V. HumRRO course was designed to meet this objective. This then is 

the first criterion and it happens to be a composite, as most terminal ob- 
jectives of self-instructional foreign language courses are. It is a composite 
criterion in the sense that there are two objectives: 



1. Are the course terminal objectives met? 



a. Does the student master the course material? 



b. Can the student after mastering the course material perform 
the actual Job given as the final course objective? 



2. The second criterion, is also a multiple one in most cases. Do 
the students used in the evaluation represent the potential student 
body? In this case, we have designated the potential student: 



a. as similar to an United States Army enlisted man. 



b. of average lan^age aptitude as measured by the Army Language 
Aptitude Test (ALAT score of 22 = 49th percentile), 

. ^ /7 -25 

c* in the„$5" to 3CT year age range^ 



d. with no relevant experience in the Slavic language family, and 

e. with no college training. 

The third criterion is simple but Important: 



3. Is the course completely self-instructional (including interim 
and final evaluation measures)? 



The next seven criteria are basically consumer oriented and are mentioned 
here primarily because they must also be taken into consideration before 
the course is constructed. Generally speaking, these seven criteria will 
vary according to the individual consumer and can only be applied in the 
case of standards laid down by the consumer which are to be met by the course 
constructor or where multiple, self-instructional courses in the same language 

w signed to meet the same terminal objectives exist arid can be compared. 

These are: 
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4. Is cost of training equipment acceptable? 

Is time for mastery of course material acceptable? 

6. Is flexibility of course scheduling acceptable? 

7. Is type of student learning environment acceptable? 

8. Is there provision for efficient maintenance and/or relearning 
of acquired foreign-language skills? 

9> Is course format generalizable to other foreign languages? 

10. Has the evaluation been replicated? 

nie HufflBRO course vas designed to meet such consumer- specified criteria 
plus the ones listed previously. After sufficient sets of course equipment 
were acquired^ seven students were selected and put through the course on 
a fully self-instructional basis. I happened to supexvise the administration 
of this course. However the actual work and contact with the students was 
carried out by an assistant who did not have any knowledge of the relevant 
foreign language. After the students had completed the course, they were 
tested to see if they had achieved course terminal objectives. Thble I lists 
the student characteristics; Table 2, the time needed to complete the course. 
Table III gives the restilts of the final end-of-course test, and Table IV 
gives the results of the job performance tests. (See pages 7, 3, 9 and 10 
for these tables.) 

A colleague. Dr. Herbert Leedy, supervised another administration of 
the same course given this time to five students. The following four tables 

represented on pages ll, 12, 13 and l4 provide information similar to that 
given previously. 

Another colleague. Dr. Catherine Garvey, constructed and evaluated a 
M a nd a r ir. Chinese course following the format of the original Russian course. 
She supervised the €uiminlstratlon of this course as given to six students. 

The results obtained were similar to the results of the two previous admin- 
istrations mentioned above. 

In summary, let us go over the ten criteria used in this evaluation of 
a self-instructional foreign language course and show in detail, where ap- 
propriate, which were or were not met. 

Criterion 1. Were the course terminal objectives met ? 

a. Did the students master the course material? 

This was measured by administering an end-of-course 
test. This test covered all of the material in the 



^ ' ?* ' ''' ' -'y " * .i;*j 







entire course and Table II gives the scores for speaking 
and understanding the foreign language and a global evaluation 
of the student's pronunciation. Each of these end-of -course 
tests vas scored by two native speakers with almost perfect 
reliability. 

Can the student perform the lactual Job given as the course 
objective? ' , __ 

V _ r 

Uhls was measured by administering a Job perfonmnce test. 

Ihe Job performance test consisted of having the student 
question a native speaker whom he had never seen before In 
a simulated combat situation. This communication situation 
was recorded for the later scoring by .native spealcers. Table 
lists the Information transmission score of each student. 

A criterion derived from the conventional standards used in evaluating 
self -ins bructional courses dealing with non- foreign language subject matter 
was used with these CONTACT courses. 



IV 



These non-language courses are hopefully aimed at achieving a 90-90 
criterion; that Is, 90 percent of the students get or ftaster 90 percent of 
the course material. We have modified this criterion to fit the special 
aspects of self- instructional foreign language training and feel this criterion 
as given below merits some consideration. 

"The self- instructional foreign language courses achieve their academic 
objective if all of the students of average or above average language-learning 
aptitude as measured by the Army Language Aptitude Test master 90 percent of 
the course material. For students with less than average language learning 
ability, academic objectives are considered achieved if JO to 80 percent of 
the course material is mastered. Such scores warrant testing the students 
to determine how well they meet Job performance objectives." 

Achievement of "job performance objectives is a much more difficult area 
of measurement and will vary considerably depending on the Job and the Job 
situation. A rough rule of thumb we have used with these CONTACT courses 
is that Job performance objectives have been achieved if students of average- 
ahd above-average language learning aptitude have been able to perform in the 
Job-simulated tests at a 90 percent measiired level of effectiveness. Students 
of below-average leuiguage learning aptitude have achieved Job performance 
objectives if they perform at a 50 percent measured level of effectiveness as 
a basic minimum. 
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Criterion 2. 



Did the students used in the evaluation represent the potential 
student body ? 

Student characteristics given in Table I for the two 
replications of course evaluation show that this criterion 
has been met, i.e., students representative of the potential 
student body have satisfactorily mastered the course. 
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Criterion 3 . Is the course completely self-instructional (including 
' interim and final evaluationi measures )? 

This criterion was met on the several levels indicated. 

First, no live instruction was given to the students; 
second, mastery of each lesson of the course was achieved 
only when every student response for that lesson was 
given correctly. Third, while interim and final tests 
were administered and scored by the researchers for 
obvious reasons, the course does have student self-scoring 
procedures available for three interim tests and the 
final end;<of- course test. 

Criterion 4. Is cost of training equipment acceptable ? 

The estimated costs x)f training equipment per man trained 
has been Judged as falling within acceptable limits from 
. the researcher's point of vlW. 

Criterion 5 • Is time for mastery of course material acceptable ? 

The time taken to master the course material by each 
student is given in Thble II and has been Judged as 
fctlling within useful limits from the researcher's 
point of view. 

Criterion 6 . Is flexibility of course scheduling acceptable ? 

The design of the course lends itself readily to a 
variety of scheduling situations. Several of the 
students in these evaluations took the course on a 
half-day schedule as compared to others who took it 
on a full-day basis. 

Criterion ?• Is type of student learning environment acceptable ? 

Any reasonably quiet study environment is sufficient to 
satisfy this criterion as was demonstrated in these 
evaluations. 

Criterion 8 . Is there provision for efficient maintenance and/or relearning 
of required foreign- language skills ? 

Problems of retention are handled in this manner. First,* 
the courses are short enough so that they may be administered 
Just prior to the activity which necessitates their admin- 
istration, thus coropletely eliminating retention problems. 
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Second^ a special set of lessons called Review-Rreview 
Lessons are so designed as to form an Integrated and 
compact unit of three or four lesson^ tapes vhlch permits 
going over all the course material on a programmed basis 
so that it is not necessary :to go through the entire 
course for relearning purposes. 

Criterion 9* Is course format generalizable to other foreign languages ? 

The construction and evaluation of the Mandarin Chinese 
course :ieets this criterion. 

Criterion 10. Has the evaluation been replicated ? 

The tvo replications of this self-instructional foreign 
language course as described in this paper satisfies this 
criterion. 

It can be seen that in order to meet these criteria meaningfully, each 
criterion must be further broken down in terms of a specific standard or 
set of conditions appropriate to the overall purpose for which the course 
is constructed. 

Regardless of any appraisal of course effectiveness that could be 
made here, the major point of this paper is that the evaluation described 
herein gives the prospective buyer enough reliable information to decide 
whether or not he will use this ..self- instructional course. 

’ Only when self-instructional foreign language courses are evaluated 
using all or most of these 10 criteria c€Ui we as programmers and producers 
of such courses^ be assured that our efforts axe not being undermined by 
a host of hastily-put-together courses offered for sale under the title of 
"Programmed, Self -instructional Language Courses." 



\ 







: . ' -■ ^" ",: -■v',' "• ■'?* ' 



■ -'ir ^ •- ■ ‘‘V- •., • .T'^ ^ •• V ■■ ■ •- 






..'*• - 






.<.4 



^>3 






- 7 - 



TABLE I 

STUDENT CHARACTERISTICS 



Army Lang. Aptitude Test°^ 



® Arny Language Aptitude Test, Office of the Adjutant General, Department of the 
Army Form- 613 I, (1957) • (The present cut-off point for acceptance at the 
Army Language School is ALAT I 8 .) 



Based on a saK 5 )le (N = 29^) of Arny Enlisted Men at Port Bragg and Fort Riley. 
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Student 


Age 


Education 


Foreign 
Lang. Exper. 


Student 
Raw Score 


■■ I j j 

\ E 

Percentile | 

Army EM^ ' 


r 


1 


18 . 


H/S, Orad. 


F5rench 


45 


97 


t 

r 

^ « , 


2 


17 


H. S. Grad. 


French 


35 


90 


? 


3 


18 


H. S. Grad. 


Spanish 


24 


60 : 

♦ 1 




4 


18 


H. S. Grad. 


Latin 


19 


34 , !| 




. 5 


■ 18 


12th Grade 


Spanish 


15 


19 


\ 


6 


17 


H. S. Grad. 


None 


14 


17 


! V 


T 


17 


12th Grade 


French 


10' 


09 ' 
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JOB FERFOBMANCE TEST 






Student 


No A of Questions 
asked by Student 


No, of Answers 
Correctly Translated 


Information Transmission 
Score 


1 


37 


35 


94 


2 


59 


;54 


91 


3 


46 


40 


86 


k 


' 50 


48 


96 


5 


5Q 


47 


94 


6 


^9 


45 


92 


7 


47 


42 


39 
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TABLE I 

Sfumm CKABACTBRISnCS 
(Second Administration) 









€l 

Amy Lang. Aptitude Test 


Student 


Aee 


Education 


Foreign 
Lans. Exner. 


Student 
Raw Score 


Percentile 
Army EM° 


1 


20 


H.S. Grad. 


Latin-Spanish 


2k 


60 


2 


22 


H.S. Grad. 


None 


17 


ITS 

CVI 


3 


20 


H.S. Grad. 


Spanish 


9 


07 


k 


18 


H.S. Grad. 


French-German 


7 


05 


.5 


21 


H.S. Grad. 


None 


00 


00 



^Arny Aptitude Test^ Office of the Adjutant General^ Department of the 

Amy Form 613 I, (195T). (The persent cut-off point for acceptance at the 
Amy Language School is ALAT lo.) 



^^ed on a (u=29l^) of Amy Enlisted Men at Fort Bragg and Fort Riley. 
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TIME FOR COMPLETION OF THE COURSE 
(Second Administration) 






student 


Working Time 
(Hours & Minutes) 


Course Time 
(Hours) 


Total Time 
(Hours) 


1 


51:36 


62 


102 


2 


32.33 


43 


78 


3 


58:06 


68 


102 


k 


. 58 : 3 ^ 


69 


96 


5 


86:30 


96 


162 
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1 


91 ■ 


98 


■ 9 ^ 


Fair 


2 


95 


98 


96 


Good 


3 


92 


98 


95 


. Excellent 


4 


95 


98 


96 


Fair 


5 


73 


75 


74- 


Fair 
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